Method for real time correlation of stereo images

ABSTRACT

In a method for correlating two stereo images, the images are subjected to a Laplacian operator and further processed to produce reduced gray scale Laplacian images in which the pixels have a value of +1, 0 or -1. Then the two images are overlapped to produce pairs of overlapping pixels. The values of the two overlapping pixels are summed in a manner so that if both pixels are +1 or both pixels are -1 the summed value is +1, if one pixel is +1 and the other pixel is -1, the resulting sum is -1 and if one or both pixel are zero, the resulting sum is zero. All of the sums or correlation values in regions about each pixel in the two overlapping images are added together to get a new correlation value for each pixel in the overlap resulting in a correlation image. Then, the two Laplacian images are shifted relative to one another and correlation values are again computed for this new overlap. This process is repeated several times resulting in correlation values for each overlap. For each pixel, the overlap which has the highest correlation value is the best match. Having determined the best match one can then determine the location of an object or point in the field of view using standard stereo processing techniques.

FIELD OF INVENTION

The invention relates to a method of comparing two stereo images todetermine a location of an object or point in a field of view.

BACKGROUND OF THE INVENTION

It is well-known that the position of an object in a volume can bedetermined using two spaced apart cameras. Both cameras take an image ofthe object at the same time or nearly the same time. Then the images arecompared to determine the location in each image of a point or series ofpoints on the object. From that information one can calculate thelocation of the object in the volume such that each point on the objecthas a known and different x, y, z coordinate.

Today there are algorithms which allow computers to perform imagematching of two stereo images. Typical images from a video cameracontain a 640×480 array of pixels. In a “black and white” image eachpixel will have a gray scale value of from 0 to 255. Current algorithmsuse the gray scale values to perform pixel comparisons to identify theposition of an object in one image with respect to the other image.Although this method is quite accurate, substantial computer capacity isneeded to perform the image matching and the process is relatively slow.As a result more expensive computer hardware is needed to do stereoimage processing. Thus, one must either use expensive image processinghardware to achieve near real time processing or be satisfied with theslow processing speeds that occur with off the shelf computingcomponents such as a personal computer (PC). Consequently, there is aneed for a method of determining the position of an object from stereoimages which is fast and can be performed on a low cost computer.

For many years the art has used the Laplacian pyramid to process andcompress images as part of stereo processing. Compressed images areeasier to store and transmit. When an image is subjected to a series ofLaplacian transforms via pyramid processing the image becomessuccessively smaller dimensionally; however, the gray scale informationremains at 8 bits. Each higher level array is half the dimensions of itspredecessor. Prior to the present invention the art used these full grayscale Laplacian images for stereo image correlation which requires muchcomputational complexity. Yet, I have found that by reducing the grayscale dimensionality of the Laplacian images I can correlate stereoimages significantly faster using a simple processor.

SUMMARY OF THE INVENTION

I provide a method for correlating two stereo images in which the imagesare subjected to a Laplacian operator to produce reduced grayscaleLaplacian images in which the pixels have a value of +1, 0 or −1. Then,I overlap the two images to produce pairs of overlapping pixels. Thevalues of the two overlapping pixels are summed in a manner so that ifboth pixels are +1 or both are −1 the summed value is +1, if one pixelis +1 and the other pixel is −1, the resulting sum is −1 and if one orboth pixels are zero the resulting sum is zero. All of the correlationvalues for the regions about the two overlapping pixels are combined toget a correlation value for the pair of pixels that overlap generating acorrelation image. Then, the two Laplacian images are shifted relativeto one another and correlation values are computed for each pair ofpixels for this particular overlap. This process is repeated severaltimes resulting in correlation images for each overlap. The overlapwhich has the highest correlation value is the best match for thatpixel. Having determined the best match for each pixel, one can thendetermine the location of an object or point in the field of view usingstandard stereo processing techniques. Other objects and advantages ofthe method will become apparent from a description of certain presentpreferred embodiments illustrated in the drawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a diagram illustrating the images used for stereo processing.

FIG. 2 is a diagram illustrating creation of correlation images byshifting Laplacian images in a vertical direction.

FIG. 3 is a diagram illustrating creation of correlation images byshifting Laplacian images in a horizontal direction.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

To correlate stereo images I begin with two images, each from one or theother of two spaced apart cameras. The first image will be called thereference image and the second image will be called the shift image. Inthe present method the original image 1 and 11 from each camera 10 issubjected to a Gaussian pyramid to produce transformed images 2, 3, 12and 13. The original gray scale images are dimensionally reduced andscaled by using the Gaussian pyramid. The original 640×480 image isreduced to a 320×240 image (level 1 Gaussian image) and further reducedto a 160×120 image (level 2 Gaussian image). The images are shown in thedrawings to contain a person's head. For purposes of illustration only asimple outline is shown. However, in actual images there would be muchgreater detail, particularly in the original image. The amount of detailbecomes progressively less for each level from the original image atlevel 0 to the level 2 Gaussian image 3 and 13. This process reduces thephysical size of the image as well as making subsequent processing morerobust with respect to sensor or other noise that may be present. Itshould be noted that the image reduction is not a necessary step forthis invention and the original image 1 and 11, could be used insubsequent steps.

Next, the Laplacian of image 3 and 13 is taken giving images 4 and 14each a 160×120 gray scale image. A Laplacian image shows the highfrequency content of the original image and is generally a signed 8 bitnumber. This Laplacian may be part of the Gaussian pyramid used togenerate images 2, 3, 12 and 13 or the Laplacian may be performed usingstandard formulae.

The next step of the process is to reduce the gray scale of image 4 and14, producing images 5 and 15, called a reduced gray scale Laplacianimage. A reduced gray scale Laplacian image is one in which each pixelwill have one of three values {−1, 0, +1 }. Values greater than or equalto 1 are assigned a value of 1, those less than or equal to −1 areassigned a value of −1 and those with a value of 0 are assigned thevalue of 0. This gray scale mapping accentuates the edges of the image.This reduced gray scale Laplacian is used to perform the correlationneeded for stereo processing.

The reduced gray scale Laplacian images 5 and 15 from each camera areoverlapped as indicated in FIGS. 2 and 3 and summed to produce acorrelation image 20 or 22. Although one could sum all pairs ofoverlapping pixels to create correlation images I have found thatsumming pixels along objects' edges of one image with correspondingpixels in the second image is sufficient. When only edge pixels aresummed correlations can be made much faster. Consequently, thecorrelation image may be derived for simpler reduced grayscale Laplacianimages. This is the reason for generating reduced grayscale Laplacianimages. In summing the pixel values of overlapping pixels if both pixelsare +1 or both pixels are −1, the summed value is +1. If one pixel is +1and the other pixel is −1, the summed value is −1. If one or both pixelshave a zero value, the sum is zero. The following matrix showscorrelation values for various pairs of pixels. $\begin{matrix}\quad & {{Image}\quad 1} \\{{Image}\quad 2} & \begin{matrix}\quad & {- 1} & 0 & {+ 1} \\{- 1} & {+ 1} & 0 & {- 1} \\0 & 0 & 0 & 0 \\{+ 1} & {- 1} & 0 & {+ 1}\end{matrix}\end{matrix}$

The correlation image 20 or 22 is also a reduced grayscale imagecontaining only the values {−1, 0, 1}. For stereo processing it isdesired to find a maximum correlation value corresponding to each pixellocation in the reference image. Thus, images 20 or 22 must be processedfurther to generate a region combination gray scale image 7. This image7 is a gray scale image which has continuous values through each pixel.Image 7 is created by combining all pixel pair values for a surroundingregion such as by using a weighted kernel. Thus, in the case of auniformly weighted kernel, all the signed numbers resulting from thecorrelation surrounding a given pixel would be added together. Oneexample of kernel size would be a 3×3 kernel, the center of the kernelwould be placed on coordinates of the pixel of interest of image 20 or22 and the correlation values would be multiplied by the correspondingkernel values to form the corresponding value in image 7.

In making the correlation the Laplacian images are overlapped severaltimes to create a set of correlation images. The various overlaps arecreated by shifting one image relative to another in a verticaldirection as indicated in FIG. 2 or by shifting one image relative tothe other image in a horizontal direction as indicated in FIG. 3. Theuse of a vertical or horizontal shift is dependent on the orientation ofthe cameras, and how they are spaced apart. In general stereo cameras 10are only displaced in one direction with their optical axes beingseparated by some distance x as indicated in FIG. 1. After correlationimages have been found for several overlaps I select for each pixel inthe non-shifted reference image the overlap image (in the format ofimage 7) having the highest correlation value as the best match. Then Ican use each overlap to determine the position of the object in thefield of view using standard stereo processing algorithms andtechniques.

This technique can be used to correlate all types of stereo images.Depending on the subject and the background greater or fewercombinations of pairs of overlapping pixels can be used to obtain acorrelation value.

Although I have described certain present preferred embodiments of mymethod, the invention is not limited thereto, but may be variouslyembodied within scope of the following claims.

I claim:
 1. A method for stereo imaging correlation comprising: a.taking a reference image of a subject with a first stereo camera thereference image comprised of a first set of pixels, each pixel having aunique coordinate and gray scale value; b. taking a shift image of thesubject with a second stereo camera, the shift image comprised of asecond set of pixels, each pixel having a unique coordinate and grayscale value, the second camera spaced from the first camera in onedimension; c. performing Laplacian transforms on the reference image andthe shift image to form a reference gray scale Laplacian image and ashift gray scale Laplacian image; d. forming a reduced gray scaleLaplacian reference image and a reduced gray scale Laplacian shift imagesuch that each pixel in the reduced gray scale Laplacian images areassigned a value of +1, 0 or −1; e. performing a correlation between thereference reduced gray scale Laplacian image and the shift reduced grayscale Laplacian image images comprising the steps of: i) overlapping thereference reduced gray scale Laplacian image and the shift reduced grayscale Laplacian image to create an overlap image having pairs ofoverlapping pixels; ii) combining the assigned values of selected pairsof overlapping pixels to form a reduced gray scale correlation image ofcorrelation values, one correlation value for each pair of overlappingpixels; iii) combining regions of the reduced gray scale correlationimage so that a continuous correlation function exists for the image;iv) shifting the shift reduced gray scale Laplacian image relative tothe reference reduced gray scale Laplacian image; and v) repeating stepsi) through iv) to create a set of gray scale overlap correlation images;f. using the gray scale overlap correlation images to select an overlapper pixel in the reference image; and g. using the selected overlap todetermine a position of at least one point on the subject.
 2. The methodof claim 1 wherein the selected pairs of overlap pixels correspond to asingle region of adjacent pixels from a first correlation image and asingle region of adjacent pixels from a second correlation image.
 3. Themethod of claim 1 wherein the selected pairs of overlap pixelscorrespond to pixels within at least two distinct regions of a firstcorrelation image and to pixels from at least two distinct regions of asecond correlation image.
 4. The method of claim 3 wherein the combiningof assigned values is performed using a non-uniform weighted kernel. 5.The method of claim 1 wherein a stereo processing algorithm is used todetermine at least one point on the subject.
 6. The method of claim 1wherein the pixels have one of three values {+1, 0, −1} and the pixelvalues are summed in a manner such that if both pixels are +1 or bothpixels are −1, the sum is +1, if one pixel is −1 and the other pixel is+1, the value is −1 and if at least one pixel is zero, the sum is zero.7. The method of claim 1 wherein the combining of assigned values isperformed using a non-uniform weighted kernel.